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Adaptive method and apparatus for automatically customizing enhanced program content to 
user preferences 



Field of the Invention 

The present invention relates to the enhancement of primary programming 
content with supplementary content, and more particularly, to a method and apparatus for 
tailoring such enhanced content to user preferences. 

Background of the Invention 

Supplemental programming content is often provided with primary 
programming material. The closed-captioning system, for example, provides a textual 
version of the audio information associated with primary programming content to permit 
people that are hearing impaired to read along with the programming content. A number of 
systems for enhancing programming content have also been proposed or suggested. 
WebTV™, for example, provides an enhanced interactive television service using Internet 
and digital technologies. For example, WebTV™ permits users to play along with game 
shows and participate in polls during television programming. 

While currently available enhanced programming services provide an 
improved viewing experience for vieXvers, they suffer from a number of limitations, which if 
overcome, could greatly expand the utility of such enhanced television systems and the 
relevance of the enhanced content they present to an individual viewer. Specifically, most 
available systems for enhancing programming content are not sufficiently customized to user 
preferences. One system for enhancing programming content is disclosed in United States 
Patent Application Serial Number 09/35 filed July 9, 1999, entitled "Method and 
Apparatus for Linking a Video Segment to Another Video Segment or Information Source," 
assigned to the assignee of the present invention and incorporated by reference herein. 

United States Patent Application Serial Number 09/372,959, filed August 1 2, 
1999, entitled "Customizing Database Information For Presentation With Media Selections," 
assigned to the assignee of the present invention and incorporated by reference herein, 
embeds codes in the primary programming content to selectively trigger enhancement or 
supplemental content in accordance with a user profile. In this manner, the user profile filters 
the enhancement material that is presented to the user. 
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In addition, United States Patent Application Serial Number 09/532,845, filed 
March 21, 2000, entitled "System and Method for Automatic Content Enhancement of 
Multimedia Output Device," hereinafter referred to as the "Automatic Content Enhancement 
System," assigned to the assignee of the present invention and incorporated by reference 
herein, utilizes video processing and feature extraction techniques to provide automatic 
content enhancement for primary programming content. The disclosed Automatic Content 
Enhancement System recognizes patterns or features in the primary programming material 
and thereafter automatically triggers the enhancement or supplemental content, optionally in 
accordance with a user profile. 

For example, a television program may include a logo for a given company 
that is recognized by the Automatic Content Enhancement System. The recognized logo is 
then correlated with enhanced content that is typically stored locally. Based on user 
preferences (for example, whether the user is interested in that particular company) and the 
correlated enhanced content, the Automatic Content Enhancement System modifies the 
primary programming content in ah appropriate way. 

For iexample, the entiairiced content presented by the Automatic Content 
Enhancement System might be a cortoercial-video clip or the telephone number of a local 
agent of the company. In one embodiment; the modification to the primary programming 
content overlays the local agent's telephone number on the video signal or buffers the 
primary broadcast signal and plays a supplemental commercial video clip. Alternatively, the 
supplemental commercial video clip can be presented using picture-in-picture technology 
simultaneously with the continued presentation of the primary programming content. 

Although these profile-based systems significantly advance the quality and 
relevance of presented erihancemeht iniaiteriaU relative to conventional techniques, they do not 
adapt to whether or not the user actually liked presented enhanced program content. A need 
therefore exists for a method arid apparatus for presenting enhanced program content in 
accordance with personalized preferences oSFirtdividual viewers that are derived from 
viewing behavior. A further need exi^-fdr ^method and apparatus for learning a viewer's 
preferences by observing the viewer's 'reactions to presented enhanced content. Yet another 
need exists for a method and apparatus that strengthens or weakens the individual inferences 
that trigger the display of enhanced program content based on viewing behavior. 
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Summary of the Invention 

Generally, a method and apparatus are disclosed for providing personalized 
supplemental programming content that enhances primary programming content. The 
enhanced program content may include, for example, biographical information about an 
individual appearing in the primary programming content or current or historical events 
regarding a geographical area indicated in the primary programming content. 

The present invention automatically and selectively triggers presentation of the 
enhanced program content in accordance with a viewer profile. The viewer profile indicates 
the level of interest, in the form of a numerical representation, of the viewer in a number of 
attributes. For example, a viewer profile might have a numerical representation indicating 
that the user is a rock fan, with the sub-attribute that the user is a particularly heavy fan of a 
certain rock band. 

According to one aspect of the invention, a viewer preference learning process 
dynamically adjusts the numerical representations set forth in the viewer profile based on the 
viewer's acceptance or rejection of enhanced program content. In this manner, the present 
invention automatically adjusts the viewer's profile by observing the viewer's reactions to 
presented enhancement content. 

The inferences contained in the viewer profile that automatically trigger the 
display of individual enhanced program content are thus strengthened or weakened based on 
the viewer's acceptance or rejection of enhanced program content, or based on an inferred 
determination as to whether the viewer enjoyed presented enhanced program content. 

A more complete understaadittg of the present invention, as well as further 
features and advantages of the present invention,' will be obtained by reference to the 
following detailed description and drawings. 

Brief Description of the Drawings 

FIG. 1 illustrates a personalized program enhancement system in accordance 
with the present invention; 

FIG. 2 illustrates a music broadcast where the content is enhanced with several 
layers of enhanced program content in accordance with the present invention; 

FIG. 3 is a table illustrating an exemplary enhanced program content database 
that classifies attributes of the enhanced program content; 

FIG. 4 A is a table illustrating an ekemplary viewer profile of FIG. 1 ; 
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FIG. 4B illustrates a Symbolicrrepresentation indicating the relative level of 
interest of a viewer in a particular program- attribute; 

FIG. 5 is a flow chart describing an exemplary personalized program 
enhancement process embodying principles of the present invention; and 

FIG. 6 is a flow chart describing an exemplary viewer preference learning 
process embodying principles of the present invention. 

Detailed Description 

FIG. 1 illustrates a personalized program enhancement system 100 in 
accordance with the present invention. The present invention seeks to enhance the enjoyment 
of a user accessing primary programming content, by providing associated, customized 
enhanced program content. The primary programming content and enhanced program 
content may comprise audio, video* textiial or other multimedia information, or a 
combination thereof. Theenh^ced^ may include, for example, biographical 

information about an individual Appearing in the primary programming content, such as a 
composer for a musical recording or Current Or historical events regarding a geographical 
area indicated in the primary programming content. 

As shown in FIG. 1, primary program content is received from a primary 
program source 110, for example, using a wireless broadcast network, such as a cellular 
telephone network, a terrestrial television broadcast network, or a digital satellite service 
(DSS) television network, or a wired network, such as the Internet, Public Switched 
Telephone Network (PSTN) or a cable television network, or a combination of the foregoing. 

Likewise, enhanced program content is received from an enhanced program 
source 120, which may be local or remote. For a remote enhanced program source 120, the 
programming content may be received by means of any wireless or wired network (or both), 
in the manner described above. It is^notai that the enhanced program content may be 
received simultaneously with the primary prdgramming content or in an off-line manner. 
The primary programming content and the erthanced program content are integrated by an 
enhancement manager 160 and presented to the viewer 195 on a display 180. 

According to another feattire of the invention, the enhancement manager 160 
can coordinate the presentation or distribution of the enhanced program content. For 
example, the enhanced program content can optionally be presented simultaneously with the 
primary programming content, or the enhanced program content may be provided, for 
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example, by electronic mail, for subsequent access. In a further variation, the primary 
content may be delayed, so that enhanced program content may be viewed first. 

The present invention automatically and selectively triggers enhanced program 
content in accordance with a viewer profile 150, discussed further below in conjunction with 
FIG. 4 A. It is noted that the viewer profile 1 50 may be associated with a specific user or a 
group of individuals, such as a household, as would be apparent to a person of ordinary skill 
in the art. According to one feature of the present invention, the personalized program 
enhancement system 100 employs a learner 170 to automatically learn the viewer's profile 
150 to greater precision by observing the viewer's reactions to presented enhancement 
content, as determined by a reaction capture module 190. The inferences contained in the 
viewer profile 1 50 that automatically trigger the display of individual enhanced program 
content can be strengthened or weakened based on the viewer's acceptance or rejection of 
enhanced program content, or based on an inferred determination as to whether the viewer 
enjoyed presented enhanced program content. These inferences can be stored in the viewer 
profile 150 thus influencing all future enhancement behavior by the system. 

Furthermore, the degree of adjustment to the inferences can vary based on the 
time of day or year, or whether the viewer selected to experience the enhanced program 
content simultaneously with the primary programming content or at a later time. For 
example, a user may have a level of interest in certain things only at certain times of the day 
or year, or when in certain moods. Likewise, ah election to receive enhanced program 
content subsequent to the presentation 6f the primary programming content may suggest a 
lower level of interest in the topic than if the usef elected immediate and simultaneous 
presentation of the enhanced program content 

In another variation, any adjustments to the inferences can be reduced or 
eliminated if the user has elected to block all enhanced program content under certain 
conditions, such as during certain categories' of programs. In other words, the fact that 
enhanced program content is ndt presented to a user under certain predefined "blocking" 
conditions, should not prevent similar enhanced program content from being presented to the 
user in the future. 

The personalized program enhancement system 100 also includes a matcher 
140 that compares characteristics of the received enhanced program content to characteristics 
specified in the viewer profile 150 to identify enhanced program content that is of interest to 
the particular viewer. Thus, while the learner 170 infers information about the viewer based 
on viewer behavior, the matcher 140 applies what has already been learned about the viewer. 
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The matcher 140 provides the personalized enhanced program content, together with the 
corresponding trigger conditions, to a triggef detector 130. 

The trigger detector 1 30, discussed further below in conjunction with FIG. 2, 
monitors the primary programming content for embedded codes or specific features, or both. 
The features in the primary prognmimirig content that trigger the display of the enhanced 
program content can be (i) embedded codes (i.e:, annotations) placed in the enhanced 
program content by or on behalf of the content provider, or (ii) features that are dynamically 
recognized in the primary programming content by the trigger detector 130. For example, the 
trigger detector 130 may employ face recognition, feature recognition, speech recognition or 
other well-known techniques to recognize predefined features in the primary programming 
content. It is noted that the embedded codes can actually be the enhanced program content, 
or a pointer to a local or remote location where the enhanced program content is stored. 

For example, the trigger detector 130 can identify a particular actor in the 
primary programming content, arid then trigger enhanced program content that is related to 
the identified actor, assuming the viewer is interested in enhanced program content about this 
actor. In one implementation, the embedded codes or extracted features are provided to the 
trigger detector 130, for example, by an iridexer (not shown),_ as a stream of tags matched to 
points or segments in the primary programming content. The trigger detector 130 then 
monitors the tags and filters out those tags that are not of interest to the user, or that do not 
correspond to available enhanced program content. 

For a more detailed discussion of a system that triggers enhanced program 
content using dynamically recognized features, see, United States Patent Application Serial 
Number 09/532,845* filed March 21, 2000, entitled "System and Method for Automatic 
Content Enhancement of Multimedia Output Device," assigned to the assignee of the present 
invention and incorporated by reference herein. For a more detailed discussion of a system 
that triggers enhanced program content using embedded codes, see, United States Patent 
Application Serial Number 09/372-959, filed August 12, 1999, entitled "Customizing 
Database Information For Presentation With Media Selections," assigned to the assignee of 
the present invention and incotpbratedby reference herein. 

Trigger Detector , 

As previously indicated; the trigger detector 130 monitors the primary 
programming content for embedded codes or specific features, or both. In an embodiment 
where the primary programming content contains iembedded enhanced program content, the 
primary programming content is manually indexed with the enhanced program content (or a 
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pointer to a file where the enhanced program content is stored). The indexing can be 
performed during content production or before content distribution, for example, by the 
content provider, the broadcaster or a third party. 

FIG. 2 illustrates a music broadcast 200 where the content 210 is enhanced 
with several layers 220, 230, 240 of enhanced program content about the same content 210. 
For example, the music broadcast 200 has been enhanced with biographical information 
about the composer in layer 220 and an explanation of the movements and passages in layer 
240. 

In an embodiment where the primary programming content is processed to 
dynamically detect predefined features that trigger the enhanced program content, the trigger 
detector 130 performs visual or audio characterization using low-level features extracted 
from the signal. For example, audio-visual characterization can be performed off-line. 
During audio-visual characterization of the content, the features are extracted and then 
classified. In this manner, the class label serVes as a trigger generator for the enhanced 
program content. 

An audio signal can be characterized by features such as: short time energy, 
band energy ratio, pause rate, arid pitch. Other features using Fourier transform and Mel 
Spectrum Frequency coefficients cah also be used. A weighted combination of these features 
can be used to characterize the audio part music program) of a video segment. The 
features are extracted using a sliding window of N milliseconds with an overlapping of M 
milliseconds (where N could be; for example, 45 and M could be 30.) The features are then 
provided to the nearest neighbor classifier to generate k classification labels for each moving 
window. Then, the k classification decisions from each window are combined to generate a 
single decision for each segment. This decision represents the identity of the music segment 
that hopefully reflects the mood, harmony and the melody itself. 

Using pitch-tb-midi conversion, the input audio signal (music) can be 
converted into a series of midi symbols, as described in A. Ghias et al., "Query by 
Humming," ACM Multimedia 95, 231-236 (1 995), incorporated by reference herein. These 
are basically part of a play list indicating which notes have been played. The melody then 
can be converted into an indexed string representing the audio segment. If the next note is 
higher than the previous one/thew it is labeled U (short for Up), if the next note is the same 
then it is labeled as S (for same), arid if the note is lower then it is labeled as D (for down). 
The opening of the Beethoven Vth symphony is then SSDUSSD. In this manner, the whole 
database is analyzed and indexed. The content provider can then search for similar strings, 
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i.e., similar music segments. If there are similar segments, then the segment is associated 
with a label (which serves as a trigger later on) of the name and location of the similar audio 
in the other video streams. When the program is played to the user, the user can see the 
related information about the matching music segments. 
MATCHER 

The matcher 140 executes a personalized program enhancement process 500, 
discussed further below in conjunction with FIG. 5, to compare features or characteristics of 
the received enhanced program content, discussed further below in conjunction with FIG. 3, 
to viewer preferences (attributes) set forth in the viewer profile 1 50, discussed further below 
in conjunction with FIG. 4A, to identify enhanced program content that is of interest to the. 
particular viewer. The matcher 140 can thus provide personalized enhanced program 
content, together with the corresponding trigger conditions, to a trigger detector 1 30. 

FIG. 3 is a table illustrating an exemplary enhanced program content database 
300 that classifies enhanced program content. As shown in FIG. 3, the enhanced program 
content database 300 contains a plurality of records 305-3 10 each associated with a different 
segment of enhanced program content. For each enhanced program content segment 
identified in column 340, the enhanced program content database 300 provides a description 
of the segment in column 345 as well as identifying attributes that indicate the target 
audience for the segment in column 350. 

Thus, as discussed further below, the matcher 140 attempts to match the 
segments of enhanced program content to the appropriate viewers, based on the attributes 
specified in the target audience column 350. It is noted that in the illustrative embodiment, 
the enhanced program content database 300 aissumes that the enhanced program content is 
stored and indexed locally . In alternate embodiments, the enhanced program content can be 
received over a network in real-time* simultaneous with the primary programming content. 
In such a real-time embodiment, the atttibutes of the target audience for the enhanced 
program content may actually be received with the enhanced program content itself, for 
example, in a parallel data stream or in header information. 

FIG. 4A is a table illustrating ah exemplary viewer profile 1 50. As shown in 
FIG. 4, the viewer profile 1 50 contains a plurality of records 405-412 each associated with a 
different user attribute or sub-attribute (hereinafter, collectively referred to as "attributes"). 
In addition, for each attribute set forth in column 440, the viewer profile 1 50 provides a 
numerical or symbolic representation in column 450, indicating the relative level of interest 
of the viewer in the corresponding attributt. For example, a viewer profile 150 might have a 
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numerical representation indicating that the user is a rock fan, with the sub-attribute that the 
user is a particularly heavy fan of the band U2. 

FIG. 4B illustrates a symbolic representation that indicates the relative level of 
interest of a viewer in a particular program attribute, such as rock music shows. As shown in 
FIG. 4B, the number of rock music shows watched by a user is translated into a discrete level 
of interest in rock music. Thereafter, if the user's level of interest in rock music is "high," for 
example, then enhanced program content corresponding to rock music will be presented to 
this user. 

The viewer profile 150 may initially be established explicitly, for example, in 
response to a questionnaire administered to the user, or implicitly, for example, by observing 
the behavior of the user. The present invention dynamically adjusts the numerical 
representations set forth in column 450 based on the viewer's reaction to presented enhanced 
program content. 

In an exemplary embodiment, the numerical representation in the viewer 
profile 150 includes an intensity scale such as: 



Number 


Description 


1 


Hates rabidly 


2 


Hates 


3 


Dislikes 


4 


Moderately negative 


5 


Neutral 


6 


Moderately positive 


7 


Likes 


8 


Loves 


9 


Loves passionately 


10 


Loves rabidly 



As previously indicated, the matcher 140 executes a personalized program 
enhancement process 500, shown in FIG. 5, to compare features of the enhanced program 
content (from FIG. 3) to viewer preferences (attributes) set forth in the viewer profile 150 
(FIG. 4) to identify enhanced program content that is of interest to the particular viewer. It is 
again noted that the illustrative embodiment of the personalized program enhancement 
process 400 assumes that the enhanced program content is stored and indexed locally for later 
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retrieval. For a discussion of an alternate, implementation that stores and indexes the 
enhanced program content locally for later retrieval, see United States Patent Application 
Serial Number 09/372,959, filed August 12, 1999, entitled "Customizing Database 
Information For Presentation With Media Selections," assigned to the assignee of the present 
invention and incorporated by reference herein. 

As shown in FIG. 5, the personalized program enhancement process 500 
initially accesses the enhanced program content database 300 during step 510 to retrieve 
attributes describing the target audience for a segment of enhanced program content. In 
addition, the personalized program enhancement process 500 accesses the viewer profile 150 
during step 520 to retrieve the numerical representations for the corresponding attributes. 

A test is performed during step 530 to determine if the retrieved numerical 
representation(s) for the corresponding attribute (s) satisfy predefined criteria. For example, 
the predefined criteria may be a minimum sebre. It is noted that the numerical or symbolic 
representation set forth in field 450 of the viewer profile 150 may be dynamically adjusted 
based on the viewer's reaction to enhanced program content in accordance with the present 
invention, as discussed hereinafter. 

If it is determined during step 530 that the retrieved numerical 
representation(s) for the corresponding attribute (s) satisfy the predefined criteria, then the 
enhanced program content is presented to the viewer together with the primary programming 
content during step 540. The presentation of the enhanced program content and the primary 
programming content is coordinated by the enhancement manager 160. If, however, it is 
determined during step 530 that the retrieved numerical representation(s) for the 
corresponding attribute (s) do not satisfy the predefined criteria, then the enhanced program 
content is not presented to the viewer ahd program control terminates during step 550. 
LEARNER - 

As previously indicated, the personalized program enhancement system 100 
employs a learner 170 to modify the viewer's profile 150 by observing the viewer's reactions 
to presented enhancement content, as determined by a reaction capture module 190. FIG. 6 is 
a flow chart describing an exemplary viewer preference learning process 600 embodying 
principles of the present invention. The viewer preference learning process 600 can 
strengthen or weaken the numerical representations corresponding to each inference 
contained in the viewer profile 150, based on the viewer's acceptance or rejection of 
enhanced program content 
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As shown in FIG. 6; the viewer preference learning process 600 initially 
monitors the viewer behavior for an acceptance or rejection of enhanced program content 
during step 610. Once it is determined during step 610 that the viewer has accepted or 
rejected some enhanced program content, program control proceeds to step 620. It is noted 
that the viewer behavior suggesting an acceptance or rejection of enhanced program content 
may be express behavior or inferred behavior. For example, the personalized program 
enhancement system 100 may require the viewer to affirmatively accept enhanced program 
content before it is presented, for example, by providing an indication of available enhanced 
program content in a small window and requiring the viewer to press a button on the remote 
control to initiate the presentation of the enhanced program content. In addition, behavior 
suggesting a positive reaction to the enhanced program content is considered an acceptance 
of the enhanced program content. For example, the following behavior suggests a positive 
reaction: expanding a smaller picture-in-picture window containing enhanced program 
content or otherwise changing the focus from the primary programming content to the 
enhanced program content. Likewise, the following behavior suggests a negative reaction: 
reducing a larger picture-in-picture window containing enhanced program content or 
otherwise changing returning the focus from the enhanced programming content to the 
primary program content. 

A test is performed during step 620 to determine if the viewer has accepted or 
rejected enhanced program content. If it is determined during step 620 that the viewer has 
accepted enhanced program content, 1 then the numerical ratings contained in the viewer 
profile 1 50 that are associated with the attributes that contributed to the selection of this 
enhanced program content are incremented during step 630. If, on the other hand, it is 
determined during step 620 that the viewer has rejected enhanced program content, then the 
numerical ratings contained in the viewer profile 150 that are associated with the attributes 
that contributed to the selection of this enhanced program content are decremented during 
step 640. 

It is noted that if a number of attributes contributed to the selection of a given 
segment of enhanced program content, the numerical representations can be adjusted in 
accordance with a sliding scale, based on the weights assigned to each contributing feature. 
Program control terminates during step 650. 

It is to be understood that the embodiments and variations shown and 
described herein are merely illustrative of the principles of this invention and that various 
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modifications may be implemented by those ^skilled, in the art without departing from the 
scope and spirit of the invention. 
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CLAIMS: 



1 • A method for updating a viewer profile (1 50) indicating preferences of a 

viewer concerning enhanced program content (120), comprising the steps of: 

presenting said enhanced program content (120) to said viewer based on 
information in said viewer profile (150); and 

updating said viewer profile (1 50) based on a reaction of said viewer to said 
enhanced program content (120). 

2. The method of claim 1, wherein said viewer profile (150) includes a numerical 
representation indicating a level of interest in at least one attribute. 

3. The method of claim 1, wherein said viewer profile (150) includes a symbolic 
representation indicating a level of interest in at least one attribute. 

4 - The method of claim 2, wherein said updating step further comprises the step 
of adjusting at least one numerical representation. 

5 - The method of claim 2, wherein said numerical representation is updated in 
accordance with a sliding scale. 

6 - The method of claim 3, wherein said updating step further comprises the step 
of adjusting at least one symbolic representation. 

7. The method of claim 1 , wherein said enhanced program content (120) is 

presented simultaneously with primary programming content (110). 



8 - The method of claim 1 , wherein said enhanced program content ( 1 20) is 

presented to said viewer asynchronously relative to said corresponding primary programming 
content (110). 
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9. The method of claim 1 , wherein said step of updating said viewer profile (150) 
further comprises the step of updating said viewer profile (1 50) based on whether said 
enhanced program content (120) is presented simultaneously with primary programming 
content (110). 

10. The method of claim 1, wherein said step of updating said viewer profile (150) 
further comprises the step of updating said viewer profile (150) based on a time of day. 

1 1 . The method of claim 1 , wherein said step of updating said viewer profile (150) 
further comprises the step of updating said viewer profile (150) based on a time of year. 

12. The method of claim 1, wherein said step of updating said viewer profile (150) 
further comprises the step of updating said viewer profile (150) based on a mood of the user. 

1 3 . The method of claim* 1 , wherein said step of updating said viewer profile (150) 
further comprises the step of updating said viewer profile (150) based on whether a 
presentation of said enhanced program content (120) is blocked under certain conditions. 

14- The method of claim 1, wherein said viewer profile (150) is associated with a 

specific individual. 

1 5 . The method of claim 1 , wherein said viewer profile (1 50) is associated with a 

group of individuals. 

1 6 - A system (1 00) for updating a viewer profile (150) indicating preferences of a 

viewer concerning enhanced program content (120), comprising: 
a memory for storing computer readable code; and 

a processor operatively coupled to said memory, said processor configured to: 
present said enhanced program content (120) to said viewer based on 

information in said viewer profile (150); and 

update said viewer profile (1 50) based on a reaction of said viewer to said 

enhanced program content (120). 
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